Predicting Resource Usage of Arbitrary Network Traffic Queries

Monitoring and mining real-time network data streams is crucial for managing and operating data networks. The information that network operators desire to extract from the network traffic is of different size, granularity and accuracy depending on the measurement task (e.g., relevant data for capacity planning and intrusion detection are very different). To satisfy these different demands, a new class of monitoring systems is emerging to handle multiple arbitrary and continuous traffic queries. Such systems must cope with the effects of overload situations due to the large volumes, high data rates and bursty nature of the network traffic – the alternative of provisioning them to handle peak rates is prohibitively expensive. In this paper, we present the design and evaluation of a system that can accurately predict the resource usage needs of network traffic queries, even in the presence of extreme and highly variable traffic conditions. The novelty of our system is that it is able to operate without any explicit knowledge of the traffic queries. Instead, it extracts a set of features from the traffic streams to build a prediction model of the query resource requirements with deterministic (and small) worst case computational cost. We present experimental evidence of the performance and robustness of our prediction methodology using real traffic traces and injecting synthetic traffic anomalies. Our results show that the system predicts the resources required to run each traffic query with small errors in all the various traffic scenarios. This prediction can later be used for load shedding purposes in order to allow current network monitoring systems to quicky react to overload situations by sampling the incoming packet streams or providing a summarized view of the input traffic.